Multimodality in Pervasive Environment

Future pervasive environments are expected to immerse users in a consistent world of probes, sensors and actuators. Multimodal interfaces combined with social computing interactions and high-performance networking can foster a new generation of pervasive environments. However, much work is still needed to harness the full potential of multimodal interaction. In this paper we discuss some short-term research goals, including advanced techniques for joining and correlating multiple data flows, each with its own approximations and uncertainty models. Also, we discuss some longer term objectives, like providing users with a mental model of their own multimodal “aura”, enabling them to collaborate with the network infrastructure toward inter-modal correlation of multimodal inputs, much in the same way as the human brain extracts a single self-conscious experience from multiple sensorial data flows. 1 Multimodal systems: Introduction Humans naturally communicate with each other in a multimodal fashion: we speak, gesture, gaze and move generating a rich, multi-streamed flow of multimedia information. Interacting with machines has traditionally been a much simpler affair, typically generating a single flow of uniform events like the discrete mouse clicks entered sequentially in a graphical user interface. As the global information infrastructure is becoming more and more pervasive, however, digital business transactions are performed in diverse situations, using a variety of mobile devices and across multiple communication channels. Rather than being forced to assume a fixed, pre-set position in front of a machine, users move freely around their work environment, starting and monitoring different transactions. Mobile terminals get Università degli Studi di Milano, Dipartimento di Tecnologie dell’informazione, e-mail: {marco.anisetti, valerio.bellandi, paolo.ceravolo, ernesto.damiani,}@unimi.it